WIP 🌱 chore(ci): Add test to check if user changes are preserved#2512
WIP 🌱 chore(ci): Add test to check if user changes are preserved#2512camilamacedo86 wants to merge 1 commit intooperator-framework:mainfrom
Conversation
✅ Deploy Preview for olmv1 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
This PR adds an end-to-end test to verify that OLMv1, when using Server-Side Apply, does not revert user-initiated changes to deployed resources. The test specifically validates the scenario where a user runs kubectl rollout restart deployment, which was problematic in OLMv0. The PR is explicitly marked as Work In Progress (WIP).
Changes:
- Added new feature file
rollout-restart.featurewith a scenario testing user-initiated deployment changes - Implemented three new step functions:
UserAddsRestartAnnotation,ResourceHasRestartAnnotation, andClusterExtensionAnnotationIsSet
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| test/e2e/features/rollout-restart.feature | New feature file defining the test scenario for verifying OLMv1 doesn't revert user-initiated changes to deployments |
| test/e2e/steps/steps.go | Added three new step functions to support the rollout restart test scenario and registered them in the step definitions |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
6173613 to
6893e06
Compare
6893e06 to
98a5126
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
98a5126 to
f626edd
Compare
f626edd to
741afe1
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
741afe1 to
91438bf
Compare
91438bf to
ef15cc9
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ef15cc9 to
add4751
Compare
add4751 to
4472c38
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
4472c38 to
a809315
Compare
a809315 to
0050274
Compare
78d26a6 to
d71302b
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
d71302b to
d411632
Compare
d411632 to
666a3cf
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
666a3cf to
2be7113
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
test/e2e/steps/steps.go
Outdated
| // ResourceHasRestartAnnotation checks that a deployment has a restart annotation. | ||
| // This confirms that user changes stay in place after OLM runs. | ||
| func ResourceHasRestartAnnotation(ctx context.Context, deploymentName string) error { |
There was a problem hiding this comment.
ResourceHasRestartAnnotation only supports Deployments (and the step text is deployment-specific). Renaming this to DeploymentHasRestartAnnotation (and updating the step binding) would make the intent clearer and avoid implying it works for arbitrary resources.
| // ResourceHasRestartAnnotation checks that a deployment has a restart annotation. | |
| // This confirms that user changes stay in place after OLM runs. | |
| func ResourceHasRestartAnnotation(ctx context.Context, deploymentName string) error { | |
| // DeploymentHasRestartAnnotation checks that a deployment has a restart annotation. | |
| // This confirms that user changes stay in place after OLM runs. | |
| func DeploymentHasRestartAnnotation(ctx context.Context, deploymentName string) error { |
72c73d1 to
d735838
Compare
d735838 to
81b15c1
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| # Wait for at least two OLM reconciliation cycles (controller runs every 10s) | ||
| And I wait for "30" seconds | ||
| # Verify user changes persisted after OLM reconciliation |
There was a problem hiding this comment.
This comment hard-codes an assumed reconciliation period (“controller runs every 10s”). If reconciliation is event-driven (e.g., triggered by the rollout restart changing the Deployment), this note is misleading and the fixed 30s sleep may be unnecessary. Consider rewording to avoid a specific cadence and/or replacing the fixed sleep with waiting for an observable reconciliation signal.
| # TODO(Fix Boxcutter): Does not work with boxcutter yet. | ||
| # Blocked by https://github.com/operator-framework/operator-controller/pull/2515 | ||
| @~BoxcutterRuntime | ||
| Scenario: User-initiated deployment changes persist after OLM reconciliation |
There was a problem hiding this comment.
The tag @~BoxcutterRuntime won’t be interpreted as “skip when BoxcutterRuntime is enabled”. CheckFeatureTags only skips when a scenario tag matches a known feature gate name and that gate is disabled; ~BoxcutterRuntime isn’t a known gate, so this scenario will still run under BoxcutterRuntime and may fail (per the TODO). Consider either removing this tag and relying on the test runner’s tag filters, or extending CheckFeatureTags to treat tags prefixed with ~ as “skip when the corresponding feature gate is enabled”.
| out, err := k8sClient("rollout", "restart", resourceName, "-n", sc.namespace) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to rollout restart %s: %w", resourceName, err) | ||
| } |
There was a problem hiding this comment.
On kubectl failure, this error drops useful diagnostics (stderr). Other steps include stderrOutput(err) in failures, which helps a lot when debugging CI flakes. Consider appending stderr (and possibly the command output) to this error message.
test/e2e/steps/steps.go
Outdated
| // If the annotation exists and has a value, it stayed in place | ||
| if out == "" { | ||
| return false | ||
| } |
There was a problem hiding this comment.
k8sClient output may contain trailing newlines/whitespace; checking out == "" can be brittle. Elsewhere in this file empty kubectl output is checked with strings.TrimSpace(out) == "". Consider trimming before the emptiness check to avoid false negatives/positives.
test/e2e/steps/steps.go
Outdated
| // ResourceHasRestartAnnotation checks that a deployment has a restart annotation. | ||
| // This confirms that user changes stay in place after OLM runs. | ||
| func ResourceHasRestartAnnotation(ctx context.Context, deploymentName string) error { | ||
| sc := scenarioCtx(ctx) |
There was a problem hiding this comment.
This step/function is deployment-specific, but the name ResourceHasRestartAnnotation reads like a generic resource assertion. Renaming it to something deployment-specific (e.g., DeploymentHasRestartAnnotation) would make failures and step registration clearer.
test/e2e/steps/steps.go
Outdated
| // Note: We wait for a fixed time (not checking in a loop) because we need to make sure | ||
| // OLM actually runs (it runs every 10 seconds). We want to check that user changes stay | ||
| // AFTER OLM runs. If we just checked in a loop, we wouldn't know if OLM ran yet. |
There was a problem hiding this comment.
The comments here assume a fixed reconciliation cadence (“OLM actually runs every 10 seconds”), but the controllers appear largely event-driven and don’t generally requeue on a fixed interval. This can mislead future readers and encourages fixed sleeps. Consider rewording to avoid stating a fixed period, and (if possible) prefer waiting on an observable reconciliation signal rather than a fixed delay.
81b15c1 to
9174a01
Compare
Add a test to ensure that OLM is not reverting user changes like kubectl rollout restart. Assisted-by: Cursor/Claude
9174a01 to
f7e96d5
Compare
Add a test to ensure that OLM is not reverting user changes like kubectl rollout restart.